System Area Networks (SANs) have been developed in which one or more processor nodes are connected to one or more I/O nodes through a fabric of switches and routers.
InfiniBand™ (IB) is an example of a system architecture providing a common I/O specification in a channel based, switched-fabric technology.
High density data links between the various processor nodes, switches, routers and I/O nodes in an IB SAN typically involve parallel transmission of data over a plurality of serial connections (“parallel-serial” links). Each link can include a number of physical lanes (e.g., copper cable or fibre optic cable). Typical multi-lane IB links can have 4, 8 or 12 physical lanes per link (denoted 4×, 8× and 12× respectively), although in principle a link may have any number of lanes.
The standard serial signalling rate is 2.5 Gbits/s (Single Data Rate, SDR) in each direction per connection. IB currently also supports double (DDR) and quad (QDR) data rates. Data in an IB system uses the industry standard 8B/10B encoding (i.e., every 10 bits carries 8 bits of data). Therefore, a 4× QDR link can carry 32 Gbits/s of useful data.
Each 10 bit entity is known as a symbol or character. Data symbols are commonly denoted Dx.y, where x ranges from 0 to 31 and y from 0 to 7. The 8B/10B standard also defines a number of control symbols, denoted by Kx.y, that can be sent in place of data symbols.
A data or link packet sent over a link is an ordered sequence of control symbols and data symbols. For example, a data packet always begins with a Start of Data Packet symbol (SDP (K27.7)) and ends with an End of Data Packet symbol (EGP (K29.7) or EBP (K30.7) depending on whether or not the packet loses integrity during transmission through the fabric). A similar system is used with link packets using SLP (K28.2) and EGP/EBP control symbols. Other control and data symbols are inserted between the SDP and EGP (or EBP) symbols, the data being byte striped across the lanes, usually sequentially.
In order to establish (or re-establish) a link between nodes over a multi-lane link it is necessary to establish the corresponding lane order and polarity of each of the lanes of the multi-lane ports at both the transmitting and receiving nodes. This is typically achieved by a process of link training. During link training an ordered set of symbols, known as a Training Sequence (TS1 or TS2) is transmitted over the link.
TS1 and TS2 contain sufficient information to enable a port receiving it to establish both the correct lane order and polarity to match the lane order and polarity of the transmitting port for each of the lanes in the link. A two way link is established (link “up”) when a port's receiver has been correctly configured (using TS1 and TS2) and the port is both receiving and transmitting Idle symbols.
A method of lane alignment in a multi-lane transmission system utilising training sequences, together with a detailed description of the InfiniBand™ system, is disclosed in U.S. Pat. No. 6,985,502 (to Bunton), the disclosure of which is incorporated here by way of reference.
Through the exchange of training sequences, such as TS1 and TS2, a link can be established for a node within an IB SAN. However, there can be occasions when it is important to monitor the flow of information through the SAN without establishing a new link. For example, an operator of a SAN may wish to monitor the performance of the SAN without disturbing the network by adding new nodes and links.
Network monitoring systems, such as Endace™'s DAG™-based system, use a passive tap to read the packets as they are transmitted over a link. While this is relatively straightforward in a single lane link, the information in a packet transmitted over a multi-lane IB link is meaningless without knowledge of the lane alignment and polarity of each lane.
One method of establishing the correct lane alignment and polarities is to wait until a training sequence is transmitted over the link. The training sequence can be readily recognised and used to set up the appropriate lane order and polarities.
However, this is not practical in general as transmission of training sequences typically only occurs when a link is being established or re-established, which can be an infrequent occurance. Monitoring of high performance computing networks, such as those using multi-lane IB technology, requires that access to the information flowing along the link be as rapid as possible. In such cases it would be an advantage to have a method of lane alignment which could be applied at any time.
It is an object of the present invention to address the foregoing problems or at least to provide the public with a useful choice.
All references, including any patents or patent applications cited in this specification are hereby incorporated by reference. No admission is made that any reference constitutes prior art. The discussion of the references states what their authors assert, and the applicants reserve the right to challenge the accuracy and pertinency of the cited documents. It will be clearly understood that, although a number of prior art publications are referred to herein, this reference does not constitute an admission that any of these documents form part of the common general knowledge in the art, in New Zealand or in any other country.
It is acknowledged that the term ‘comprise’ may, under varying jurisdictions, be attributed with either an exclusive or an inclusive meaning. For the purpose of this specification, and unless otherwise noted, the term ‘comprise’ shall have an inclusive meaning—i.e. that it will be taken to mean an inclusion of not only the listed components it directly references, but also other non-specified components or elements. This rationale will also be used when the term ‘comprised’ or ‘comprising’ is used in relation to one or more steps in a method or process.
Further aspects and advantages of the present invention will become apparent from the ensuing description which is given by way of example only.